Image display device

ABSTRACT

An image display device may include a communication unit that receives frame images sequentially transmitted from image pickup devices, a display unit that displays the frame images, a storage unit that stores processible information on an amount of the frame images and priority information indicating priorities of the frame images, an acquisition unit that acquires status information, a determination unit that determines a data amount of the frame images transmitted by each of the image pickup devices within a predetermined period of time based on the processible information and the priority information stored in the storage unit and the status information acquired by the acquisition unit, and a control unit that performs control that causes information corresponding to the data amount determined by the determination unit to be transmitted from the communication unit to each of the plurality of image pickup devices.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image display device that receivesframe images from a plurality of image pickup devices and displays thereceived frame images.

Priority is claimed on Japanese Patent Application No. 2011-097128,filed Apr. 25, 2011, the content of which is incorporated herein byreference.

2. Description of the Related Art

All patents, patent applications, patent publications, scientificarticles, and the like, which will hereinafter be cited or identified inthe present application, will hereby be incorporated by reference intheir entirety in order to describe more fully the state of the art towhich the present invention pertains.

Japanese Unexamined Patent Application, First Publication No.2001-356753 discloses a technique of reducing an amount of informationon a network when a plurality of videos are transmitted and received viathe network. In a display control system disclosed in JapaneseUnexamined Patent Application, First Publication No. 2001-356753, whenimages are displayed on a plurality of display areas on a screen of animage display device based on video signals from a plurality of signalsources, respectively, image display attribute information and displaysignal selection information of each display area are transmitted toeach signal source from the image display device, and, based on theresolution of the display area added to the image display attributeinformation received at a signal source side, the video signal, in whichinformation content is reduced in advance, is transmitted to the imagedisplay device from each signal source. Thereby, it is possible toreduce an amount of information on a network.

When the plurality of videos are transmitted/received, if the amount ofinformation of the plurality of transmitted videos exceeds an amount ofprocessible information at the reception side, since deletion, or thelike, of the frame of the videos displayed on the image display devicemay occur, the amount of information of the transmitted videos may bereduced, and a processing load of the reception side may be reduced. Theabove-mentioned display control system reduces the amount of informationof the video signal from the signal source based on resolution of theplurality of display areas on the screen of the image display device.Since the display control system reduces the amount of information ofthe video signal from the signal source based on the resolution of thedisplay area, with regard to the video displayed on the display areahaving a low resolution, even in the video signal in which a user of thedisplay control system requires the amount of information, only thevideo in which the amount of information has been reduced is displayed.

SUMMARY

The present invention provides an image display device capable ofreducing a processing load related to an image display and performingimage display in which image priority is reflected.

An image display device may include: a communication unit configured toreceive frame images sequentially transmitted from each of a pluralityof image pickup devices; a display unit configured to display the frameimages from each of the plurality of image pickup devices that have beenreceived by the communication unit; a storage unit configured to storeprocessible information on an amount of the frame images that can beprocessed within a predetermined period of time, and store priorityinformation indicating priorities of the frame images previouslydetermined with respect to a photographing operation, a photographingenvironment, or a state of a photographed subject related to the frameimages that have been received by the communication unit; an acquisitionunit configured to acquire status information indicating thephotographing operation, the photographing environment, or the state ofthe photographed subject for each of the image pickup devices; adetermination unit configured to determine a data amount of the frameimages transmitted by each of the image pickup devices within apredetermined period of time based on the processible information andthe priority information stored in the storage unit and the statusinformation acquired by the acquisition unit; and a control unitconfigured to perform control that causes information corresponding tothe data amount determined by the determination unit to be transmittedfrom the communication unit to each of the plurality of image pickupdevices.

The acquisition unit may acquire the status information based on datatransmitted from the image pickup device with the frame image.

The image display device may further include: a remote operation unitconfigured to instruct the photographing operation with respect to theimage pickup device. The acquisition unit may acquire the statusinformation based on the photographing operation instructed by theremote operation unit.

The image display device may further include: an analysis unitconfigured to analyze a displacement amount of the photographed subjectbased on the frame image. The acquisition unit acquires the statusinformation based on a result of the analysis by the analysis unit.

The status information may be information on a framing operation of theimage pickup device.

The status information may be information on a sound volume detected bythe image pickup device.

The status information may be information on whether the subject presentin the frame image is recognizable.

The status information may be information on a movement amount in theframe image of the subject present in the frame image.

According to the present invention, since a data amount of the frameimage transmitted by each of the image pickup devices within apredetermined period is determined based on processible information,priority information and status information, the processing load relatedto the image display can be reduced and image display in which the imagepriority is reflected can be performed.

BRIEF DESCRIPTION OF THE DRAWINGS

The above features and advantages of the present invention will be moreapparent from the following description of certain preferred embodimentstaken in conjunction with the accompanying drawings, in which:

FIGS. 1A to 1E are system configuration diagrams illustrating a controlmethod in accordance with a first preferred embodiment of the presentinvention;

FIG. 2 is a block diagram illustrating a configuration of a displayterminal in accordance with the first preferred embodiment of thepresent invention;

FIG. 3 is a block diagram illustrating a configuration of an imagepickup device in accordance with the first preferred embodiment of thepresent invention;

FIG. 4 is a flowchart illustrating a sequence of operations of a displayterminal in accordance with the first preferred embodiment of thepresent invention;

FIG. 5 is a reference view illustrating a data format of frame rateinformation in accordance with the first preferred embodiment of thepresent invention;

FIG. 6 is a reference view illustrating a data format of frame ratechange information in accordance with the first preferred embodiment ofthe present invention;

FIG. 7 is a reference view illustrating a storage format of image pickupdevice displacement data in accordance with the first preferredembodiment of the present invention;

FIG. 8 is a view for explaining a method of determining a frame rate inaccordance with the first preferred embodiment of the present invention;

FIG. 9 is a reference view for explaining the method of determining theframe rate in accordance with the first preferred embodiment of thepresent invention;

FIG. 10 is a reference view illustrating a priority table in accordancewith the first preferred embodiment of the present invention;

FIG. 11 is a flowchart illustrating a sequence of operations of theimage pickup device in accordance with the first preferred embodiment ofthe present invention;

FIGS. 12A and 12B are block diagrams illustrating configurations of adisplay terminal and an image pickup device of a first variant inaccordance with the first preferred embodiment of the present invention;

FIG. 13 is a block diagram illustrating a configuration of a displayterminal of a second variant in accordance with the first preferredembodiment of the present invention; and

FIG. 14 is a block diagram illustrating a configuration of a displayterminal of a third variant in accordance with the first preferredembodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention will be now described herein with reference toillustrative preferred embodiments. Those skilled in the art willrecognize that many alternative preferred embodiments can beaccomplished using the teaching of the present invention and that thepresent invention is not limited to the preferred embodimentsillustrated for explanatory purpose.

Hereinafter, a first preferred embodiment of the present invention willbe described with reference to the accompanying drawings. FIGS. 1A to 1Eshow examples of a control method of a display control system equippedwith an image pickup device and a display terminal (an image displaydevice) in accordance with the first preferred embodiment of the presentinvention. In FIG. 1, a display terminal 101 receives and displaysreal-time videos transmitted from image pickup devices 102 and 103,respectively. In the control method in accordance with the firstpreferred embodiment of the present invention, based on image pickupdevice displacement data added to frame rate change informationillustrating a change instruction of a frame rate and a maximumprocessible amount of the display terminal 101, the frame rate (a dataamount of the frame image transmitted by each image pickup device withinone frame) of the real-time video transmitted to the display terminal101 from each of the image pickup devices is controlled, and thus, aload of the display terminal 101 is reduced.

In addition, the frame rate change information in accordance with thefirst preferred embodiment of the present invention is informationtransmitted from the image pickup devices 102 and 103 (hereinafter, inthe first preferred embodiment, this will be referred to as simply an“image pickup device” unless it indicates a specific image pickupdevice) to the display terminal 101, and information including at leastimage pickup device displacement data. Further, the image pickup devicedisplacement data in the first preferred embodiment is data indicating astate of a photographing situation or a state of a photographingenvironment of each image pickup device, and a state of a subjectcaptured by the image pickup device.

In addition, a photographing situation of the image pickup device in thefirst preferred embodiment means a situation (state) of a framingoperation (photographing manipulation) such as pan and tilt, focus, andzoom-wide manipulations, a state of a photographing environment means adisplacement amount of a sound volume in surroundings, and a state of asubject means whether the subject is recognized or not (whether thesubject is recognizable or not), and a movement amount (a displacementamount), or the like, in the frame image of the subject present in theframe image.

Further, a maximum processible amount in the first preferred embodimentmeans a maximum number of frame images for which reception-displayprocess is possible by the display terminal 101 within a predeterminedtime. Furthermore, a real-time video in the first preferred embodimentmeans a video photographed by the image capturing unit of the imagepickup device, and a video configured by data of a frame image of eachframe, which is generated in time sequence, and sequentially stored by astorage unit of the image pickup device.

In the examples shown in FIGS. 1A to 1E, examples of a control method ina situation in which a real-time video is transmitted from the imagepickup devices 102 and 103 to the display terminal 101 with a frame ratedetermined by the display terminal 101, based on the number of terminalsof the image pickup device and the maximum processible amount of thedisplay terminal 101, are shown. In this situation, when the framingoperation is performed in the image pickup device 102, frame rate changeinformation is transmitted from the image pickup device 102 to thedisplay terminal 101. Based on the image pickup device displacement dataadded to the frame rate change information transmitted from the imagepickup device 102 and the maximum processible amount, the frame rate ofeach of the image pickup devices is re-determined by the displayterminal 101. As the frame rate information indicating the re-determinedframe rate of each of the image pickup devices is transmitted to theimage pickup devices 102 and 103 from the display terminal 101, theframe rate of the real-time video transmitted from the image pickupdevices 102 and 103 to the display terminal 101 is controlled at theframe rate represented by the frame rate information.

In addition, the frame rate information in the first preferredembodiment is information transmitted to the image pickup devices 102and 103 from the display terminal 101, and information including atleast information that designates the frame rate of the real-time videotransmitted from the image pickup devices 102 and 103 to the displayterminal 101.

An example shown in FIG. 1A represents a situation in which the framerate information determined by the display terminal 101 is transmittedto the image pickup devices 102 and 103 from the display terminal 101based on the number of terminals of the image pickup device and themaximum processible amount of the display terminal 101. An example shownin FIG. 1B represents a situation in which the real-time video istransmitted at the frame rate indicating the frame rate informationreceived by the display terminal 101 from the image pickup devices 102and 103. An example shown in FIG. 1C represents a situation in which theframing operation is performed in the image pickup device 102, and theframe rate change information is transmitted from the image pickupdevice 102 to the display terminal 101.

An example shown in FIG. 1D represents a situation in which the framerate information re-determined by the display terminal 101 based on theimage pickup device displacement data added to the frame rate changeinformation received in FIG. 1C is transmitted to the image pickupdevice from the display terminal 101. An example shown in FIG. 1Erepresents a situation in which the real-time video is transmitted atthe frame rate represented with the frame rate information re-receivedby the display terminal 101 from the image pickup devices 102 and 103.

In addition, in the examples of FIGS. 1A to 1E, since the framingoperation is performed by the image pickup device 102 only, while theframe rate of the image pickup device 102 is set to be high, the imagepickup device in which the frame rate is set to be high is not limitedto only one terminal. For example, when the framing operation isperformed by the plurality of image pickup devices, there may be aplurality of image pickup devices in which the frame rate is set to behigh, and when there is no image pickup device in which the framingoperation is performed, the state shown in FIG. 1B may be maintained.

Further, in the examples of FIGS. 1A to 1E, while the frame rate changeinformation is transmitted from the image pickup device 102 to thedisplay terminal 101 under the condition that the framing operation ofthe image pickup device 102 is performed, the conditions under which theframe rate change information is transmitted are not limited to theabove condition. For example, the frame rate change information may betransmitted when there is a variation in determination of whether thesubject is recognized by image pickup device, a movement amount in theframe image of the subject present in the frame image, or a change insound volume detected by the image pickup device.

FIG. 2 shows of a configuration of the display terminal 101 inaccordance with the first preferred embodiment of the present invention.Hereinafter, the configuration of the display terminal 101 will bedescribed with reference to FIG. 2. The display terminal 101 inaccordance with the first preferred embodiment includes a display unit201, a determination unit 202, an acquisition unit 203, a storage unit204, a processing unit 205 (a control unit), and a communication unit206.

The display unit 201 displays a frame image received by thecommunication unit 206. The determination unit 202 determines a framerate of each of the image pickup devices based on the maximumprocessible amount, the number of image pickup devices that transmit thereal-time videos, some or all of the image pickup device displacementdata, and the priority information for determining priority of the imagepickup devices.

The acquisition unit 203 analyzes the frame rate change informationreceived by the communication unit 206, and acquires the image pickupdevice displacement data of each of the image pickup devices. Thestorage unit 204 stores the frame image, the maximum processible amount,the number of image pickup devices that transmit the real-time videos,the frame rate change information, the frame rate information, and apriority table. In addition, the priority table in accordance with thefirst preferred embodiment of the present invention is previouslydetermined with respect to all data that configures the image pickupdevice displacement data, and is a table illustrating the weight of thepriority of the frame image. The processing unit 205 controls theoperation of the display terminal 101. The communication unit 206performs communication with the image pickup device.

FIG. 3 shows a configuration of the image pickup device in accordancewith the first preferred embodiment of the present invention. Theconfiguration of the image pickup device will be described with respectto FIG. 3. The image pickup device in accordance with the firstpreferred embodiment of the present invention includes an imagecapturing unit 301, a processing unit 302, an acquisition unit 303, astorage unit 304, and a communication unit 305.

The image capturing unit 301 generates the frame image obtained byphotographing the subject, and acquires the real-time video. Theprocessing unit 302 controls the operation of the image pickup device.The acquisition unit 303 performs acquisition of the image pickup devicedisplacement data and generation of the frame rate change information.The storage unit 304 stores the real-time video, the frame rateinformation, the number of image non-transmissions, and a maximumobtainable amount. The communication unit 305 performs communicationwith the display terminal 101. In addition, the number of imagenon-transmissions is the number of times to skip a transmission processfor frame images to be transmitted from the image pickup device to thedisplay terminal 101. In addition, the maximum obtainable amount inaccordance with the first preferred embodiment of the present inventionis information indicating the frame rate of the real-time video obtainedby the image pickup device.

FIG. 4 shows the operation of the display terminal 101 in accordancewith the first preferred embodiment of the present invention.Hereinafter, the operation of the display terminal 101 will be describedwith reference to FIG. 4.

The processing unit 205 starts a real-time video display process when areal-time video display start command is received (step S401). Inaddition, the real-time video display start command in accordance withthe first preferred embodiment of the present invention is a commandthat allows the display terminal 101 to start the real-time videodisplay process, and a command that is issued under a condition in whichthe image pickup device and the display terminal 101 belong to the samenetwork. The condition under which the command is issued is not limitedto the above condition. For example, the command may be issued under thecondition that information related to an instruction of command issuancebe received from the image pickup devices belonging to the same networkvia the communication unit 206, or the command may be issued under thecondition that a user interface unit further included in theconfiguration of the display terminal shown in FIG. 2 and the command isissued by a user input.

The processing unit 205 determines the maximum processible amount whenthe real-time video display process is started, and stores the amount inthe storage unit 204 (step S402). The processing unit 205 issues a framerate information generation command to the determination unit 202 afterstorage of the maximum processible amount.

In addition, while the maximum processible amount in accordance with thefirst preferred embodiment of the present invention is a valuedetermined by the processing unit 205, the information pre-stored in thestorage unit 204 may be used. Further, for example, provided that acommunication speed is X and a data amount of one frame of the frameimage that configures the real-time video is Y, the maximum processibleamount in accordance with the first preferred embodiment of the presentinvention may be obtained by calculating X/Y. Furthermore, the framerate information generation command in accordance with the firstpreferred embodiment of the present invention is a command issued to thedetermination unit 202 from the processing unit 205, and a command thatallows the determination unit 202 to generate the frame rateinformation.

When the frame rate information generation command is received from theprocessing unit 205, the determination unit 202 determines the framerate of the real-time video transmitted to the display terminal 101 fromeach of the image pickup devices based on the number of image pickupdevices that transmit the real-time videos and the maximum processibleamount stored in the storage unit 204, and stores the determined framerate as the frame rate information in the storage unit 204 (step S403).The determination unit 202 issues a frame rate information storagetermination notice that notifies the processing unit 205 of completionof storage of the frame rate information after storage of the frame rateinformation. In addition, the number of image pickup devices thattransmit the real-time videos can be obtained from the information(beacon, etc.) transmitted from the image pickup devices under thecondition that the image pickup devices and the display terminal 101belong to the same network.

Further, the frame rate information is information configured in a dataformat shown in FIG. 5. The data format of the frame rate informationwill be described later. Furthermore, the method of determining theframe rate in step S403 will be described later.

The processing unit 205 transmits the frame rate information stored inthe storage unit 204 to each of the image pickup devices via thecommunication unit 206 when the frame rate information storagetermination notice is received from the determination unit 202 (stepS404). The processing unit 205 transmits an image transmission startcommand to each of the image pickup devices via the communication unit206 after transmission of the frame rate information (step S405). Inaddition, the image transmission start command in accordance with thefirst preferred embodiment of the present invention is a command tostart transmission of the real-time video to the image pickup device.

The processing unit 205 is in a reception standby state in whichreception of the frame image transmitted from each of the image pickupdevices is on standby, after transmission of the image transmissionstart command (step S406). The processing unit 205 displays the frameimage on a predetermined region of the display unit 201 when the frameimage is received from the image pickup device via the communicationunit 206 (step S407). The processing unit 205 issues a frame rate changeinformation reception start command to the acquisition unit 203 afterdisplay of the frame image. In addition, the frame rate changeinformation reception start command in accordance with the firstpreferred embodiment of the present invention is a command issued to theacquisition unit 203 from the processing unit 205, and a command tostart reception processing of the frame rate change information to theacquisition unit 203.

The acquisition unit 203 is in a reception standby state in which thereception of the frame rate change information is on standby, when theframe rate change information reception start command is received (stepS408). The acquisition unit 203 performs the processing shown in stepS411 when the frame rate change information is not received for acertain amount of time, and acquires the image pickup devicedisplacement data from the received frame rate change information andstores the data in the storage unit 204, when the frame rate changeinformation is received via the communication unit 206. The acquisitionunit 203 issues a frame rate change information reception terminationnotice to the determination unit 202 after storage of the image pickupdevice displacement data. In addition, the frame rate change informationreception termination notice in accordance with the first preferredembodiment of the present invention is a notice issued to thedetermination unit 202 from the acquisition unit 203, and a notice thatallows the determination unit 202 to re-determine the frame rateinformation.

In addition, the frame rate change information is information configuredin a data format shown in FIG. 6. After the acquisition unit 203acquires the image pickup device displacement data, as shown in FIG. 7,only the image pickup device displacement data is stored in the storageunit 204. The data format of the frame rate change information and astorage scheme for the image pickup device displacement data will bedescribed later.

When the frame rate change information reception termination notice isreceived from the processing unit 205, the determination unit 202re-determines the frame rate of the real-time video transmitted to thedisplay terminal 101 from each of the image pickup devices based on theimage pickup device displacement data stored in the storage unit 204 andthe maximum processible amount, and stores the determined frame rate asthe frame rate information in the storage unit 204 (step S409). Thedetermination unit 202 issues the frame rate information storagetermination notice to the processing unit 205 after storage of the framerate information. In addition, the method of determining the frame ratein step S409 will be described later.

When the frame rate information storage termination notice is received,the processing unit 205 transmits the frame rate information stored inthe storage unit 204 to each of the image pickup devices via thecommunication unit 206 (step S410). After the frame rate information istransmitted or when the frame rate change information is not receivedfor a certain amount of time in step S408, the processing unit 205determines whether a real-time video display end command is issued ornot (step S411). The processing unit 205 performs the processing shownin step S412 when the real-time video display end command is issued, andperforms re-processing from the reception processing of the frame imageshown in step S406 when the real-time video display end command is notissued.

In addition, the real-time video display end command in accordance withthe first preferred embodiment of the present invention is a command toend the real-time video display process performed by the displayterminal 101, and a command issued before the image pickup device andthe display terminal 101 are disconnected. The condition under which thecommand is issued is not limited the above-mentioned condition, but, forexample, the command may be issued under the condition that theinformation related to the command issuance instruction from the imagepickup device, which transmits the real-time video, be received via thecommunication unit 206, and the command may be issued under thecondition that a user interface unit is further included in theconfiguration of the display terminal shown in FIG. 2 and the command isissued by a user input.

When the real-time video display end command is received, the processingunit 205 transmits the image transmission end command to each of theimage pickup devices and secedes from the network (step S412). Inaddition, the image transmission end command in accordance with thefirst preferred embodiment of the present invention is a commandtransmitted to each of the image pickup devices from the displayterminal 101, and a command to end transmission of the real-time videoto the image pickup device.

The data format of the frame rate information shown in step S403, or thelike, will be described with reference to FIG. 5. The frame rateinformation is information constituted by a message type 501, a datalength 502, a transmission destination IP address 503, a transmissionsource IP address 504, and a frame rate definition value 505.

Data indicating that the information is the frame rate information isstored in the message type 501. A data length of the frame rateinformation is stored in the data length 502. An IP address of the imagepickup device that designates the frame rate is stored in thetransmission destination IP address 503. An IP address of the displayterminal 101 that has transmitted the frame rate information is storedin the transmission source IP address 504. A definition value of theframe rate of the frame image transmitted from the image pickup deviceto the display terminal 101 is stored in the frame rate definition value505.

The data format of the frame rate change information shown in step S408,or the like, will be described with reference to FIG. 6. The frame ratechange information is information constituted by a message type 601, adata length 602, a transmission destination IP address 603, atransmission source IP address 604, and image pickup device displacementdata 605.

Data indicating that the information is the frame rate changeinformation is stored in the message type 601. A data length of theframe rate change information is stored in the data length 602. An IPaddress of the display terminal 101 is stored in the transmissiondestination IP address 603. An IP address of the image pickup devicethat has transmitted the frame rate change information is stored in thetransmission source IP address 604. Photographing displacement data ofthe image pickup device corresponding to the transmission source IPaddress 604 is stored in the image pickup device displacement data 605.

A method of storing the image pickup device displacement data shown instep S408 will be described with reference to FIG. 7. The information ofthe image pickup device displacement data stored in the storage unit 204is constituted by an image pickup device displacement data search table701, an IP address 702, and image pickup device displacement data 703.

The image pickup device displacement data search table 701 is associatedwith the IP address 702 of the image pickup device that has acquired theimage pickup device displacement data 703 retained in the storage unit204. The IP address 702 is associated with the image pickup devicedisplacement data 703 of a corresponding image pickup device. The imagepickup device displacement data 703 include a state or a displacementamount of the framing operation such as panning and tilting or zooming,a displacement amount of a sound volume, a state of subject recognition,and so on. As described above, the image pickup device displacement datasearch table 701 is stored in the storage unit 204 in a state in whichthe IP addresses 702 assoicated with the image pickup devicedisplacement data 703 are associated with one another.

The method of determining the frame rate shown in step S403 will bedescribed with reference to FIG. 8. In step S403, determination of theframe rate of each of the image pickup devices is performed by evenlyallocating the number of frames with respect to each of the image pickupdevices based on the number of connected image pickup devices and themaximum processible amount stored in the storage unit 204. For example,in FIG. 8, provided that the maximum processible amount of the displayterminal 101 is 30 fps, since the number of image pickup devices isthree, that is, image pickup devices 802, 803 and 804, 10 fps isallocated to each of the image pickup devices such that the sum of theframe rates of the image pickup devices does not exceed the maximumprocessible amount of the display terminal 101.

The method of determining the frame rate shown in step S409 will bedescribed with reference to FIGS. 8 to 10. In step S409, determinationof the frame rate of each of the image pickup devices is performed byindividually allocating the number of frames with respect to each of theimage pickup devices based on the maximum processible amount stored inthe storage unit 204 and the priority of each of the image pickupdevices determined from the image pickup device displacement data storedin the storage unit 204.

In the first preferred embodiment, the priority of each of the imagepickup devices is determined by a photographing step of each of theimage pickup devices predicted from the displacement data of each of theimage pickup devices. Alternatively, the priority of each of the imagepickup devices is determined by comparing the displacement data of eachof the image pickup devices with a priority table shown in FIG. 10.

In step S409, when determination of the priority of each of the imagepickup devices is performed in the photographing step of each of theimage pickup devices, the photographing step of each of the image pickupdevices is predicted from the displacement data of each of the imagepickup devices. For example, in FIGS. 8 and 9, the image pickup device802 can determine the situation in which the subject 801 is in an angleof view and a zooming operation is performed from the image pickupdevice displacement data 901 of the image pickup device 802. For thisreason, it is determined that the image pickup device 802 is in a stepof composition determination.

In addition, the image pickup device 803 can determine the situation inwhich the subject 801 is not in the angle of view and panning andtilting are performed from the image pickup device displacement data 902of the image pickup device 803. For this reason, the image pickup device803 is predicted to be in a step in which capturing of the subject 801is performed. Further, the image pickup device 804 can determine thesituation in which the subject 801 is in the angle of view and theframing operation is not performed from the image pickup devicedisplacement data 903 of the image pickup device 804. For this reason,the image pickup device 804 is predicted to be in a step in which thephotographing can be performed.

After prediction of the photographing step of each of the image pickupdevices, determination of the priority of each of the image pickupdevices is performed. In the situation in which the framing operation isperformed, movement of the subject in the real-time video is increasedby movement of the image pickup device by panning and tilting, or thelike. In the situation in which movement of the subject in the real-timevideo is large and the frame rate of the real-time video is low, a usercannot easily follow the subject and fine adjustment of the framingoperation becomes difficult. For this reason, during the framingoperation, i.e., in the step in which the capturing of the subject isperformed, determination of high priority can be considered so that theframe rate becomes high. For example, in FIGS. 8 and 9, when thepriority is determined to be high for a step farthest from the step inwhich the photographing can be performed, the image pickup device 803 inthe step in which the capturing of the subject is performed has a highpriority, the image pickup device 802 in the step of compositiondetermination has an ordinary priority, and the image pickup device 804in the step in which the photographing can be performed has a lowpriority.

In addition, since the information (priority information) indicating arelationship between each photographing step and priority is previouslystored in the storage unit 204, after prediction of the photographingstep, the priority is determined to correspond to the photographing stepbased on the information stored in the storage unit 204. Further, in thefirst preferred embodiment, while the priority is determined to be highfor the step farthest from the step in which the photographing can beperformed, the method of determining the priority based on thephotographing step is not limited to the above-mentioned method. Forexample, a method of determining the priority as high for the stepnearest to the step in which the photographing can be performed or amethod of determining the priority as high at the image pickup device inthe step of composition determination may be provided.

After determination of the priority, allocation of the number of framesof each of the image pickup devices is performed based on the maximumprocessible amount stored in the storage unit 204 and the priority ofeach of the image pickup devices. In addition, the weight of thepriority when the priority is determined by the photographing step ofeach of the image pickup devices is allocated as a natural number in asequence from the image pickup device having a low priority, andallocation of the number of frames is performed based on the maximumprocessible amount and the weight of the priority.

For example, in FIGS. 8 and 9, the image pickup device 804 having a lowpriority becomes priority 1, the image pickup device 802 having anordinary priority becomes priority 2, and the image pickup device 803having a high priority becomes priority 3. For this reason, providedthat the maximum processible amount of the display terminal 101 is 30fps, 10 fps is allocated to the image pickup device 802 having anordinary priority, 15 fps is allocated to the image pickup device 803having a high priority, and 5 fps is allocated to the image pickupdevice 804 having a low priority, so that the sum of the frame rates ofthe image pickup devices does not exceed the maximum processible amountof the display terminal 101.

Meanwhile, in step S409, the case in which the priority of each of theimage pickup devices is determined by comparing the displacement data ofeach of the image pickup devices with the priority table (priorityinformation) shown in FIG. 10 is as follows. As shown in FIG. 10, anitem of data configuring the image pickup device displacement data,detailed information belonging to the item, and the priority allocatedto the detailed information thereof are associated with the prioritytable.

Items of the data configuring the image pickup device displacement dataare framing, subject recognition, and so on. The detailed informationbelonging to the framing is pan, tilt, zoom, wide, focus, and so on. Thedetailed information belonging to the subject recognition is“unrecognizable” indicating that the subject cannot be recognized, “inmotion” indicating that the subject is in motion, and “not in motion”indicating that the subject is not in motion. The others may be a changein sound volume, or the like.

As described above, in the situation in which the framing operation isperformed, movement of the subject in the real-time video is increasedby movement of the image pickup device by the pan and tilt, or the like,and fine adjustment of the framing operation becomes difficult. For thisreason, the priority with respect to the pan or tilt is set to be a highvalue. In addition, since movement of the subject in the real-time videois also generated by the zooming operation, or the like, the prioritywith respect to the zoom, wide, and focus is set to be a next highestvalue of the priority with respect to the pan and tilt.

Meanwhile, in the subject recognition, in order for the user to easilyfind the subject when the subject cannot be recognized, the prioritywith respect to “unrecognizable” is set to be a high value. In addition,in order for the user to easily find the subject even when the subjectis in motion, the priority with respect to “in motion” is set to be thenext highest value of the priority with respect to “unrecognizable.”

In determination of the priority of each of the image pickup devicesusing the priority table, comparison of the displacement data of each ofthe image pickup devices with the priority table is performed, and thesum of the priority of the item corresponding to the image pickup devicedisplacement data shown in the priority table is determined as thepriority of each of the image pickup devices. After determination of thepriority, allocation of the number of frames of each of the image pickupdevices is performed based on the maximum processible amount stored inthe storage unit 204 and the priority of each of the image pickupdevices. In addition, when the priority of each of the image pickupdevices is determined using the priority table, allocation of the numberof frames is performed based on the maximum processible amount and theweight of the priority.

For example, in FIGS. 8, 9 and 10, when the displacement data of each ofthe image pickup devices is compared with the priority table, thepriority of each of the image pickup devices is as follows. In addition,the image pickup device 802 corresponds to the image pickup devicedisplacement data 901, the image pickup device 803 corresponds to theimage pickup device displacement data 902, and image pickup device 804corresponds to the image pickup device displacement data 903. Thepriority of the image pickup device 802 becomes 7(=5+1+1), the priorityof the image pickup device 803 becomes 21(=10+1+10), and the priority ofthe image pickup device 804 becomes 2(=1+1). For this reason, providedthat the maximum processible amount of the display terminal 101 is 30fps, 7 fps are allocated to the image pickup device 802, 21 fps areallocated to the image pickup device 803, and 2 fps are allocated to theimage pickup device 804, so that the sum of the frame rate of each ofthe image pickup devices does not exceed the maximum processible amountof the display terminal 101.

In addition, while the priority table shown in FIG. 10 has been used asthe priority table in accordance with the first preferred embodiment ofthe present invention, the priority table is not limited to FIG. 10. Forexample, in addition to variation of each framing operation, a movementamount of the subject, change in sound volume, etc., the priority tablemay have different weights of priorities according to the respectivevariations.

FIG. 11 shows an operation of the image pickup device in accordance withthe first preferred embodiment of the present invention. Hereinafter,the operation of the image pickup device will be described withreference to FIG. 11.

The processing unit 302 starts an image transmission process when animage transmission process start command is received (step S1101). Inaddition, the image transmission process start command in accordancewith the first preferred embodiment of the present invention is acommand that allows the image pickup device to start the imagetransmission process, and a command issued under the condition that thedisplay terminal 101 and the image pickup device belong to the samenetwork. The condition under which the command is issued is not limitedto the above-mentioned condition, but, for example, the condition may bea condition that a command is issued by receiving information related toa command issuance instruction via the communication unit 305 from thedisplay terminal 101 belonging to the same network, or a condition thata user interface unit is further included in the configuration of theimage pickup device shown in FIG. 3 and the command is issued by userinput.

The processing unit 302 is in a reception standby state in whichreception of the frame rate information is on standby after a start ofthe image transmission process (step S1102). The processing unit 302determines the maximum obtainable amount when the frame rate informationis received via the communication unit 305, and stores the frame rateinformation and the maximum obtainable amount in the storage unit 304(step S1103). The processing unit 302 determines a defined value of thenumber of image non-transmissions, after storage of the frame rateinformation and the maximum obtainable amount, based on the frame rateinformation and the maximum obtainable amount, and stores the value inthe storage unit 304 (step S1104).

As described above, the maximum obtainable amount is informationindicating the frame rate of the real-time video acquired by the imagepickup device. In addition, the number of image non-transmissions is thenumber of times to skip a transmission process of frame images to betransmitted from the image pickup device to the display terminal 101.Further, a defined value of the number of image non-transmissions is avalue calculated based on a definition value of the frame rate added tothe frame rate information and the maximum obtainable amount. Forexample, provided that a definition value of the frame rate is 5 fps andthe maximum obtainable amount is 30 fps, when 30 fps is changed with 5fps, since the frame image may be transmitted once at 6 frames, thedefined value of the number of image non-transmissions is calculated as6. Furthermore, separately from the defined value of the number of imagenon-transmissions, the number of image non-transmissions is stored inthe storage unit 304, and a value after clear is set to the number ofimage non-transmissions.

The processing unit 302 is in a reception standby state in whichreception of the image transmission start command after storage of thedefined value of the number of image non-transmissions (step S1105).When the image transmission start command is received via thecommunication unit 305, the processing unit 302 transmits the latestframe image of the real-time video obtained by the image capturing unitand sequentially stored in the storage unit 304 to the display terminal101 via the communication unit 305 (step S1106).

After transmission of the frame image in step S1106, the processing unit302 determines whether the number of image non-transmissions stored inthe storage unit 304 is equal to the defined value of the number ofimage non-transmissions stored in the storage unit 304 (step S1107). Theprocessing unit 302 performs processing shown in step S1108 when thenumber of image non-transmissions is equal to the defined value of thenumber of image non-transmissions, and performs processing shown in stepS1110 when unequal to the defined value. When the number of imagenon-transmissions processed in step S1107 is equal to the defined valueof the number of image non-transmissions, the processing unit 302transmits the latest frame image of the real-time video stored in thestorage unit 304 to the display terminal 101 via the communication unit305 (step S1108), and clears the number of image non-transmissions (stepS1109). After clearance of the number of image non-transmissions, theprocessing unit 302 issues an image pickup device displacement dataacquisition command to the acquisition unit 303. In addition, the imagepickup device displacement data acquisition command in accordance withthe first preferred embodiment of the present invention is a commandissued to the acquisition unit 303 from the processing unit 302, and acommand to start a photographing situation of the image pickup devicewith respect to the acquisition unit 303 or acquisition of thedisplacement amount of the subject in photographing.

Meanwhile, when the number of image non-transmissions processed in stepS1107 is not equal to the defined value of the number of imagenon-transmissions, the processing unit 302 counts up the number of imagenon-transmissions (step S1110). After the number of imagenon-transmissions is counted up, the processing unit 302 issues theimage pickup device displacement data acquisition command to theacquisition unit 303.

In step S1109 or step S1110, when the image pickup device displacementdata acquisition command is received from the processing unit 302, theacquisition unit 303 performs acquisition of the image pickup devicedisplacement data and determines whether the acquired photographingsituation or the displacement amount of the subject is varied or not(step S1111). In addition, the image pickup device displacement dataacquired by the acquisition unit 303 is stored in the storage unit 304,and in step S1111, determination is performed by comparing thephotographing situation or the displacement amount of the subjectacquired from the finally acquired image pickup device displacement datawith the photographing situation or the displacement amount of thesubject acquired from the image pickup device displacement data acquiredone time before the finally acquired image pickup device displacementdata.

When there is a variation in a photographing situation or displacementamount of the subject, the acquisition unit 303 adds the image pickupdevice displacement data to the frame rate change information and storesthe added result in the storage unit 304, and issues a frame rate changeinformation storage termination notice to the processing unit 302.Meanwhile, when there is no variation in a photographing situation ordisplacement amount of the subject, processing shown in step S1115 isperformed. In addition, the frame rate change information storagetermination notice in accordance with the first preferred embodiment ofthe present invention is a command issued to the processing unit 302from the acquisition unit 303, and a command that allows transmission ofthe frame rate change information to the processing unit 302.

When the frame rate change information storage termination notice isreceived, the processing unit 302 transmits the frame rate changeinformation stored in the storage unit 304 to the display terminal 101via the communication unit 305 (step S1112). After transmission of theframe rate change information, the processing unit 302 is in thereception standby state in which reception of the frame rate informationis on standby (step S1113). When the frame rate information is receivedvia the communication unit 305, the processing unit 302 stores thereceived frame rate information in the storage unit 304, and performsprocessing shown in step S1114.

After storage of the frame rate information received in step S1113, theprocessing unit 302 updates the defined value of the number of imagenon-transmissions based on the frame rate information received in stepS1113 and stored in the storage unit 304 and the maximum obtainableamount stored in the storage unit 304 in step S1103 (step S1114).

After the defined value of the number of image non-transmissions isupdated in step S 1114 or when it is determined that there is novariation in a photographing situation or displacement amount of thesubject in step S1111, the processing unit 302 determines whether theimage transmission end command is received from the display terminal 101(step S1115). When the image transmission end command is received, theprocessing unit 302 stops transmission of the real-time video to thedisplay terminal 101 and ends the image transmission process. Meanwhile,when the image transmission end command is not yet received, theprocessing unit 302 performs processing again from determination of thenumber of image non-transmissions shown in step S1107.

Effects According to the Preferred Embodiment

The preferred embodiment of the present invention is as described above.According to the preferred embodiment of the present invention, since adata amount of the frame image transmitted from each of the image pickupdevices is determined such that the sum of the data amount (frame rate)of the frame image transmitted from each of the image pickup devicesdoes not exceed the maximum processible amount of the display terminal,a processing load related to the image display in the display terminalcan be reduced. In addition, based on the photographing operationrelated to the frame image, the photographing environment, the imagepickup device displacement data indicating a state of the photographedsubject, and information related to the priority, as a high data amountis primarily allocated to the image pickup device in a state in whichthe user requires information amount, image display in which thepriority of the image is reflected can be performed such that theinformation amount of the video in which the user requires theinformation amount is not reduced.

In addition, since the information related to the framing operation ofthe image pickup device is included in the image pickup devicedisplacement data, based on the framing operation such as pan and tilt,zoom-wide, focus, etc., in each of the image pickup devices, the dataamount of the frame image transmitted to the display terminal from eachof the image pickup devices can be determined.

Further, since the information related to the sound volume detected bythe image pickup device is included in the image pickup devicedisplacement data, based on the peripheral sound volume detected by eachof the image pickup devices, the data amount of the frame imagetransmitted to the display terminal from each of the image pickupdevices can be determined.

Furthermore, since the information related to whether the subjectpresent in the frame image is recognized is included in the image pickupdevice displacement data, based on whether the subject present in theframe image is recognized, the data amount of the frame imagetransmitted to the display terminal from each of the image pickupdevices can be determined.

In addition, since the information related in the movement amount in theframe image of the subject present in the frame image is included in theimage pickup device displacement data, based on the movement amount ofthe subject present in the frame image, the data amount of the frameimage transmitted to the display terminal from each of the image pickupdevices can be determined.

(Variant)

While the preferred embodiment of the present invention has beendescribed as above, the present invention is not limited to thepreferred embodiment of the present invention.

(First Variant)

For example, as shown in FIG. 12A, the display terminal 101 may furtherinclude a remote operation unit 1201 configured to instruct thephotographing situation in the image pickup device so that thephotographing situation of the image pickup device can be instructedfrom the display terminal 101. The remote operation unit 1201 isconfigured to receive an operation related to the photographingsituation from the user, and the processing unit 205 transmits theinformation of the photographing situation received by the remoteoperation unit 1201 from the user to the image pickup device via thecommunication unit 206. In the image pickup device, the processing unit302 receives the information of the photographing situation transmittedfrom the display terminal 101 via the communication unit 305, andcontrols a state of the image capturing unit 301 based on the receivedinformation of the photographing situation.

In addition, in the system represented as a first variant, as shown inFIG. 12B, based on variation in a photographing situation of the imagepickup device in which the remote operation unit 1201 is treated as aportion of the acquisition unit 203 and performed using the remoteoperation unit 1201, allocation of the number of frames of each of theimage pickup devices in the display terminal 101 may be performed. Theremote operation unit 1201 is configured to receive an operation relatedto the photographing situation from the user, and the acquisition unit203 acquires the image pickup device displacement data from theinformation of the photographing situation received by the remoteoperation unit 1201 from the user. Further, when the display terminal101 is configured as shown in FIG. 12B, the image pickup device may nothave the acquisition unit 303.

(Second Variant)

For example, as shown in FIG. 13, the acquisition unit 203 of thedisplay terminal 101 may further include a video analysis unit 1301configured to analyze the frame image transmitted from the image pickupdevice, so that allocation of the number of frames of each of the imagepickup devices in the display terminal 101 is performed based onvariation of the subject acquired by the video analysis unit 1301through analysis of the frame image. The acquisition unit 203 acquiresthe image pickup device displacement data from the variation of thesubject acquired by the video analysis unit 1301 through analysis of theframe image. In addition, when the display terminal 101 is configured asshown in FIG. 13, the image pickup device may not have the acquisitionunit 303.

(Third Variant)

For example, as shown in FIG. 14, the acquisition unit 203 in thedisplay terminal 101 may further include the remote operation unit 1201shown in FIG. 12 and the video analysis unit 1301 shown in FIG. 13, sothat allocation of the number of frames of each of the image pickupdevices in the display terminal 101 is performed based on variation in aphotographing situation of the image pickup device performed using theremote operation unit 1201 and variation of the subject analyzed by thevideo analysis unit 1301. In addition, when the display terminal 101 isconfigured as shown in FIG. 14, the image pickup device may not have theacquisition unit 303.

While preferred embodiments of the present invention have been describedand illustrated above, it should be understood that these are examplesof the present invention and are not to be considered as limiting.Additions, omissions, substitutions, and other modifications can be madewithout departing from the scope of the present invention. Accordingly,the present invention is not to be considered as being limited by theforegoing description, and is only limited by the scope of the claims.

1. An image display device comprising: a communication unit configuredto receive frame images sequentially transmitted from each of aplurality of image pickup devices; a display unit configured to displaythe frame images from each of the plurality of image pickup devices thathave been received by the communication unit; a storage unit configuredto store processible information on an amount of the frame images thatcan be processed within a predetermined period of time, and storepriority information indicating priorities of the frame imagespreviously determined with respect to a photographing operation, aphotographing environment, or a state of a photographed subject relatedto the frame images that have been received by the communication unit;an acquisition unit configured to acquire status information indicatingthe photographing operation, the photographing environment, or the stateof the photographed subject for each of the image pickup devices; adetermination unit configured to determine a data amount of the frameimages transmitted by each of the image pickup devices within apredetermined period of time based on the processible information andthe priority information stored in the storage unit and the statusinformation acquired by the acquisition unit; and a control unitconfigured to perform control that causes information corresponding tothe data amount determined by the determination unit to be transmittedfrom the communication unit to each of the plurality of image pickupdevices.
 2. The image display device according to claim 1, wherein theacquisition unit acquires the status information based on datatransmitted from the image pickup device with the frame image.
 3. Theimage display device according to claim 1, further comprising: a remoteoperation unit configured to instruct the photographing operation withrespect to the image pickup device, wherein the acquisition unitacquires the status information based on the photographing operationinstructed by the remote operation unit.
 4. The image display deviceaccording to claim 1, further comprising: an analysis unit configured toanalyze a displacement amount of the photographed subject based on theframe image, wherein the acquisition unit acquires the statusinformation based on a result of the analysis by the analysis unit. 5.The image display device according to claim 1, wherein the statusinformation is information on a framing operation of the image pickupdevice.
 6. The image display device according to claim 1, wherein thestatus information is information on a sound volume detected by theimage pickup device.
 7. The image display device according to claim 1,wherein the status information is information on whether the subjectpresent in the frame image is recognizable.
 8. The image display deviceaccording to claim 1, wherein the status information is information on amovement amount in the frame image of the subject present in the frameimage.